[AMD] Optimize kimik2.5-fp4-mi355x-vllm-disagg: image bump + serve sync + all-TP4 P/D (EP off) / 优化 kimik2.5-fp4-mi355x-vllm-disagg：镜像升级 + serve 参数对齐 + 全 TP4 P/D（关闭 EP）#2247
Conversation
…h all-TP4 config Update the changelog description to match the final config: TP8 dropped everywhere (real-weight sweep showed no per-GPU benefit), 1P(TP4)/1D(TP4) covers the full curve, 1P(TP4)/2D(TP4) adds decode KV headroom for the KV-bound 8k/1k tail (conc 256/512), EP off. Note ~2-3x tok/s/GPU vs the prior 1P2D TP8/EP8 baseline (PR #1585). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # perf-changelog.yaml
…2247 Co-authored-by: Cursor <cursoragent@cursor.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🟡
perf-changelog.yaml:4800— The new perf-changelog.yaml entry for kimik2.5-fp4-mi355x-vllm-disagg has 'pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX' — the placeholder was never replaced with the actual PR number (2247). Every other entry in the file uses a real PR number, so this should be updated before merge to avoid a dead/wrong link.Extended reasoning...
The bug: the newly-added perf-changelog.yaml entry for
kimik2.5-fp4-mi355x-vllm-disaggends with:pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
XXXis a literal, unfilled placeholder rather than a real PR number. Per the PR metadata, this change is PR #2247, so the intended value ishttps://github.com/SemiAnalysisAI/InferenceX/pull/2247.Why this happened / why nothing catches it:
perf-changelog.yamlis a plain data file with no schema validation or CI check enforcing thatpr-linkcontains a numeric PR reference — it's free-form text, so a placeholder string is syntactically valid YAML and passes any lint/parse step. The convention of using a real PR number is purely a human convention established by every prior entry in the file, not something enforced in code.Proof / concrete walkthrough:
- Look at the five entries immediately preceding this one in the diff context: they use
pull/2197,pull/2198,pull/2222, etc. — all real, resolvable PR numbers. - The new entry for
kimik2.5-fp4-mi355x-vllm-disagg(added in this PR) instead haspull/XXX. - The actual PR is #2247 (per PR metadata: title '[AMD] Optimize kimik2.5-fp4-mi355x-vllm-disagg...').
- Once this PR merges, anyone reading the changelog and clicking that link gets a 404 (GitHub has no PR literally numbered "XXX"), whereas every other row in the file resolves to the actual PR that introduced the change.
Impact: This is confined to documentation/metadata — it does not affect any benchmark config, runner behavior, or CI. It breaks the changelog's traceability convention (each entry should let a reader jump straight to the PR that made the change) and would need a separate follow-up commit to fix if merged as-is.
Fix: Replace
XXXwith2247in thepr-linkfield before merging:pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2247
All three verifiers independently confirmed this via direct diff inspection and agreed on nit severity — none refuted it, and there's no reasonable argument that
XXXis intentional given the unbroken pattern of real PR numbers throughout the rest of the file. - Look at the five entries immediately preceding this one in the diff context: they use
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29522769567 |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29547977874 |
|
cc @seungrokj @chunfangamd for helping out signing off |
|
vllm recipe update: vllm-project/recipes#650 |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29547977874
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Range: 96.89%-97.27%, link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29547977874
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Not apply (For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.)
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
vllm-project/recipes#650
Signed: @chunfangamd
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29615689753 |
3 participants
Description
Retunes the
kimik2.5-fp4-mi355x-vllm-disaggdisaggregated config for MI355X, and fixes twobugs that were failing the sweep. Every engine change below was A/B'd locally on real weights
(gfx950, TP4, 8k/1k, on the pinned CI image) before being adopted.
Recipe (
configs/amd-master.yaml,amd_utils/models_vllm.yaml)Image bump to
vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9.All-TP4 P/D layout (TP8 dropped) — a real-weight sweep showed TP8 decode gives no per-GPU
benefit over TP4.
1P(TP4)/1D(TP4)covers the full curve;1P(TP4)/2D(TP4)adds decode KVheadroom for the KV-bound 8k/1k tail (conc 256/512). All layouts keep prefill+decode ≤ 3 nodes.
Expert parallelism off (
ep:1) — a single-node TP8 real-weight sweep showed EP is 14–27% slower.Serve-flag/env sync with the single-node recipe (
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4,HSA_NO_SCRATCH_RECLAIM=1, AITER defaults).Dropped the
cudagraph_mode: PIECEWISEpin (carried over from bring-up [AMD] vLLM Kimi MXFP4 & MiniMax M2.5 FP8 disaggregated prefill-decode for MI355X #1569), taking vLLM'sdefault
FULL_AND_PIECEWISEas the single-node recipe does. Measured vs the PIECEWISE pin, output tok/s:PIECEWISE loses at every concurrency. The two full modes tie within run-to-run noise, so the
default wins the tiebreak by also matching single-node. The gap decays to ~0 by conc 128.
Added
--kv-cache-dtype fp8,--max-model-len 32768,--max-num-seqs 256,--max-num-batched-tokens 32768. Measured vs the flags above, total tok/s:TTFT −18% and TPOT −22% at conc 128. fp8 KV does most of the work on two counts: it halves KV
read bandwidth (a win at every concurrency, not only when KV-bound) and doubles resident capacity
(1,358,885 → 2,730,007 tokens), relieving the KV-bound decode tail. No accuracy cost: local
TP4 GSM8K scored 0.9697 flexible / 0.9712 strict — identical to the bf16-KV run and clear of the
0.90 gate in
utils/evals/thresholds.json.Bug fix: the multi-node eval never produced results (
amd_utils/server_vllm.sh)server_vllm.shnever setEVAL_MAX_MODEL_LEN, sorun_lm_evalfell back to its 16384 default andderived
max_tokens = 16384 − 4096 = 12288— against engines served with--max-model-len 9472.Every gsm8k request was rejected. From run 29522769567:
Node-0 exits 1 → "Launch multi-node job script" fails → nothing staged → "Upload eval results" and
"Verify eval scores" fail as consequences, and
eval_results_allcomes back[]. This is why thethroughput jobs passed while both eval arms failed: throughput never touches that path.
Now
EVAL_MAX_MODEL_LENis derived viasetup_eval_context(ISL+OSL+256, capped at the model'snative max) and clamped to the engine's served
--max-model-len, mirroring the single-node recipe.A pre-flight probe at the real output budget also runs before the 1319-question eval, so a rejected
token budget fails in seconds with the engine's own error instead of burning ~20 minutes of 3-node
time over 4203 retries. Both fixes apply to every
*-vllm-disaggconfig, not just kimik2.5.Bug fix: the pinned vllm-router image was garbage-collected (
amd_utils/job.slurm)vllm/vllm-routerretains only ~16 nightlies on Docker Hub; the pinnednightly-20260629-e667ebbnow 404s, so rank 0'sdocker run -dfails with "manifest unknown"after the allocation is already up. Re-pinned to
nightly-20260716-1fbcde7.Results
The perf gain against the last disagg run on main is real and large: ~2.0–2.3x throughput per GPU at matched interactivity, plus the curve now reaches into a low-latency region the old config couldn't touch at all.
Test plan
full-sweep-fail-fastgreen forkimik2.5-fp4-mi355x-vllm-disagg.中文说明
针对 MI355X 重新调优分离式(disagg)配置
kimik2.5-fp4-mi355x-vllm-disagg，并修复了两个导致扫描(sweep)失败的缺陷。下述所有引擎参数改动，均已在本地以真权重(gfx950、TP4、8k/1k、CI 固定镜像)
完成 A/B 实测后才予以采用。
配方改动（
configs/amd-master.yaml、amd_utils/models_vllm.yaml）vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9。1P(TP4)/1D(TP4)覆盖完整并发曲线；
1P(TP4)/2D(TP4)为受 KV 限制的 8k/1k 高并发尾部（并发 256/512）提供额外解码 KV 容量。所有布局的预填充+解码节点数 ≤ 3。
ep:1)：单节点 TP8 实测显示 EP 相比 dense 慢 14%~27%。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4、HSA_NO_SCRATCH_RECLAIM=1、AITER 默认值）。cudagraph_mode: PIECEWISE固定值（自 bring-up [AMD] vLLM Kimi MXFP4 & MiniMax M2.5 FP8 disaggregated prefill-decode for MI355X #1569 沿用至今），改用 vLLM 默认的FULL_AND_PIECEWISE，与单节点配方一致。相较 PIECEWISE，输出吞吐提升见上表：PIECEWISE 在所有并发点均落后；两种 full 模式的差异在运行间噪声范围内，因此默认值凭借"同时与单节点保持一致"
胜出。该优势在并发 128 时衰减至约 0。
--kv-cache-dtype fp8、--max-model-len 32768、--max-num-seqs 256、--max-num-batched-tokens 32768：相较上述参数，总吞吐在并发 16/64/128 分别提升+16.2%/+19.1%/+27.4%，并发 128 时 TTFT 降低 18%、TPOT 降低 22%。其中 fp8 KV 贡献最大，
原因有二：将 KV 读取带宽减半（在所有并发下均有收益，而非仅在受 KV 限制时），以及将常驻容量
翻倍（1,358,885 → 2,730,007 tokens），从而缓解受 KV 限制的解码尾部。精度无损失：本地 TP4
GSM8K 得分 0.9697（flexible）/ 0.9712（strict），与 bf16 KV 结果一致，且远高于
utils/evals/thresholds.json中 0.90 的门槛。缺陷修复：多节点评估始终无法产出结果（
amd_utils/server_vllm.sh）server_vllm.sh从未设置EVAL_MAX_MODEL_LEN，导致run_lm_eval回退到默认的 16384，并推导出max_tokens = 16384 − 4096 = 12288——而引擎实际以--max-model-len 9472启动，因此每个 gsm8k请求都被拒绝（详见上方英文部分引用的 run 29522769567 日志）。node-0 以 rc=1 退出，导致
"Launch multi-node job script" 步骤失败、评估产物未暂存，"Upload eval results" 与
"Verify eval scores" 随之连带失败，
eval_results_all返回[]。这也解释了为何吞吐任务全部通过而两个评估任务均失败：吞吐路径根本不会走到这段代码。
现在改为通过
setup_eval_context推导EVAL_MAX_MODEL_LEN（ISL+OSL+256，上限为模型原生最大值），并按引擎实际提供的
--max-model-len进行钳制，与单节点配方保持一致。此外，在执行 1319 道题的完整评估之前，会先以真实输出预算发送一次预检请求：一旦 token 预算被拒绝，可在数秒内带着引擎自身
的错误信息失败，而不是在 4203 次重试中白白消耗约 20 分钟的三节点机时。以上两项修复对所有
*-vllm-disagg配置均生效，不限于 kimik2.5。缺陷修复：固定的 vllm-router 镜像已被回收（
amd_utils/job.slurm）vllm/vllm-router在 Docker Hub 上仅保留约 16 个 nightly 版本；此前固定的nightly-20260629-e667ebb现已返回 404，导致 rank 0 在资源已分配之后才因"manifest unknown" 拉取失败。现重新固定至
nightly-20260716-1fbcde7。结果